Stability of a neural network model with small-world connections 
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Small-world networks are highly clustered networks with small distances among the nodes. There 
are many biological neural networks that present this kind of connections. There are no special 
weightings in the connections of most existing small-world network models. However, this kind of 
simply-connected models cannot characterize biological neural networks, in which there are different 
weights in synaptic connections. In this paper, we present a neural network model with weighted 
small-world connections, and further investigate the stability of this model. 

PACS numbers: 89.75.Hc, 87.18.Sn, 84.35.-K, 05.45.-a, 
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A great deal of research interest in the theory and ap- 
plications of small- world networks have arisen [1-8] since 
the pioneering work of Watts and Strogatz [9]. Some 
common properties of complex networks, such as Inter- 
net servers, power grids, human communities, and disor- 
dered porous media, are mainly determined by the way 
of connections among their vertices or nodes. Among 
various networks, one extremal case is a regular network 
with a high degree of local clustering and a large aver- 
age distance, while the other extremal case is a random 
network with negligible local clustering and a small av- 
erage distance. In between the two extremes there are 
small-world networks, which are a special type of com- 
plex networks with a high degree of local clustering as 
well as a small average distance. 

Many biological neural networks are small-world net- 
works [10-13]. In most existing literature about small- 
world networks, there are no weightings in their internal 
connections of nodes. However, there are many networks, 
particularly biological neural networks, having weights 
associated with the connections. These connection- 
weighted networks cannot be described and characterized 
by those previously proposed small- world network mod- 
els. Of particular interest is [13], where small- world neu- 
ral networks have random weights in their connections, 
which studied the cluster coefficient and the characteris- 
tic path of such networks. In this paper, we use dynam- 
ical equations to describe a connection-weighted small- 
world neural network model, and then further study its 
stability with respect to the network topology. 

For this purpose, consider a neural network with N 
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neurons described by 

du{t) _ 
dt ~ 



Au(t) + Wg{u{t)) + I 



(1) 



where u(t) — [ui(i), u%(t), ■ ■ ■ , ujy(t)] is the neuron 
state vector, A — diag{ai,<22, ■ • ■ ,cin} is a positive di- 
agonal matrix, g(u) = [g-i (u 1 ),g 2 {u 2 ), • ■ ■ ,gN(uN)] T de- 
notes the neuron activation functions with g(0) = 0,1 = 
[Jij/a,--- ,In] T is a constant vector, W = {wyjjvxAf 
is the connection- weighting matrix, in which, similar to 
[13], Wij is defined as follows: if there is a connection 
between neuron i and neuron j (j ^ i), then there is 
a uniform random distribution = Wji in the con- 
nection, with values < lUy = Wji < 1; otherwise, 
w ij — w ji = (i 7^ i) ■ The diagonal elements of W are 
all zeros, which means that there are no self-connection 
of nodes within the network. Throughout this paper, 
we assume that each activation function in (1) satisfies 
the following sector condition: There is a real constant, 
k G R, such that 



< 



9j(x) -gjjy) 
x-y 



<k, Vx,yeR, j = l,2,. 
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This type of neural networks with full and regular 
connections has been extensively investigated. However, 
neural networks with small-world connections have not 
been thoroughly studied, particularly with respect to 
their stabilities. For example, it is not clear whether 
or not the small-world neural networks are easier to be 
stabilized than the fully connected ones. In this paper, 
we address this question by carefully studying model (1). 

In the following, we always shift the equilibrium u* 
of network (1) to the origin. By making the transform 
x(t) = u(t) — u* , we convert model (1) to the following: 



dx(t) 
dt 



-Ax{t) + Wf(x(t)) 



(2) 
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where fj( Xj (t)) = gj { Xj {t)+u*)- gj {u*), j = 1, 2, • • • , N. 
Note that fj also satisfies a sector condition in the form 
of 

fj(*i) (fi(xj) - kxj) < 0, j = 1, 2, • • • , N (3) 

For this small-world neural network model, we have 
the following theoretical result: 

Lemma: Let X ma x (M) denote the largest eigenvalue of 
matrix M. If X m ax (~j + W) < 0, then network (2) is 
asymptotically stable about the origin. 

Proof. Select a Lyapunov function as 

V(x(t)) = / fi{*)ds 

Using the method presented in [14, 15], it is easy to verify 
that V(x(t)) is a Lyapunov function. In fact, if we define 
the following function: 

Gi(u) = min fi{6)d6, j " /i(fl)dfl| 

Then we have G,(0) = 0,Gi(u) = Gi(\u\); for r e R + , 
Gi(r) > 0,r > 0;G;(r) -> +oo, r -> +oo. Let G = 
mm{G t }. Then 

n^) =EiLi/ x '/i(*)d« 

>EtiG«(^) 

= EL^(N) 

>EtiG(k|) 
>G(N) 

Therefore, we have a lower bound achieved by a positive, 
radially unbounded function. It is then easy to verify [15] 
that 

G(\x(t)\)<V(\x(t)\)<qk\\x\\ 2 ,q>l 

The derivative of V(x) along the trajectories of (2) is, 
by using (3), 

V(t) = ih(xi), ■■■ , In{xn)] [xi, ■ ■ ■ ,x N ] T 

= f T (x(t)) \-Ax{t) + Wf{x{t))] 

= -f T (x(t))Ax(t)+f T (x(t))Wf(x(t)) 

< f T (x(t))[-± + W]f(x(t)) 

< X max (-i + W)\\f(x(t))\\ 2 

Therefore, if \ max (— 4 + W) < 0, then we have 
V(x(t)) < 0, implying that network (2) is asymptotically 
stable about the origin. 

Because A is a diagonal positive matrix, it is easy to 
deduce the following Corollary. 

Corollary: If \ max (W) < minj^f}, then network (2) is 
asymptotically stable about the origin. 

Although we can derive some less conservative stability 
conditions for (2), we only use the above results in this 



paper, because they are very simple and easy to verify. 
Further, because these conditions use only the maximum 
eigenvalue of the connection matrix W, we can "average" 
them when using statistical methods to investigate the 
properties of the connection matrix, as further explained 
in the following. 

Aiming to describe a transition from a regular net- 
work to a random network, [9] introduced an interest- 
ing model, now referred to as the small-world (SW) net- 
work. The original SW model can be described as follows. 
Take a one-dimensional lattice of N vertices arranged in 
a ring with connections only in between nearest neigh- 
bors. We then "rewire" each connection with probability 
p. Rewiring in this context means reconnecting randomly 
the whole lattice, with the constraint that no two differ- 
ent vertices can have more than one connection in be- 
tween, and no vertex can have a connection with itself. 

Note, however, that it is quite possible for the SW 
model to be broken into unconnected clusters. This prob- 
lem can be resolved by a slight modification of the SW 
model, suggested by Newman and Watts (NW) lately 
[1]. In the NW model, we do not break any connec- 
tion between any two nearest neighbors. Instead, we 
add with probability p a connection between each uncon- 
nected pair of vertices. Likewise, we do not allow a vertex 
to be coupled to another vertex more than once, or a ver- 
tex to be coupled with itself. For p = 0, it reduces to the 
originally nearest-neighbor coupled network; for p = 1, it 
becomes a globally coupled network. Here, we are inter- 
ested in the NW model starting from a nearest-neighbor 
lattice with 4-neighbors and a connection-adding proba- 
bility < p < 1. 

From a coupling-matrix point of view, network (2) 
with small-world connections evolves according to the 
rule that, in the nearest-neighbor coupling matrix W, 
if Wij = 0, we set Wij — Wji — w with probability p and 
a uniformly randomly distributed weight < w < 1. Wc 
denote the new small- world coupling matrix by W(p, N) 
and let \ ma x (p, N) be its largest eigenvalue. According 
to Corollary 1, if X max (p, N) < min{^f}, then the cor- 
responding small-world neural network is asymptotically 
stable about its zero state. 

Clearly, the network stability depends on the proba- 
bility p, so it is more practical to investigate the statis- 
tical properties of the connection matrix W. It is easy 
to see that the mathematical expectation of the num- 
ber of neurons that are connected to each neuron, i.e., 
the number of nonzero entries in each row of W(p, N), is 
n c = 4+ (N — 5)p. Because of the uniform random distri- 
bution of the weight values, it is also easy to see that the 
mathematical expectation of the sum of entries in each 
row of W(p, N) is 0.5[4 + (N — 5)p], where N > 5 (the 
smallest neuron number of the nearest-neighbor lattice 
with 4-neighbors). Thus, by Lemma 2 of [16], we can 
calculate the mathematical expectation of X m ax (j>, N) , 
which is 0.5[4+(A r — 5)p], where N > 5. Hence, the small- 
world neural network is asymptotically stable about its 
zero state, in the sense of mathematical expectation, if 
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0.5[4 + (N — 5)p\ < min{^-}, N > 5. This also means 
that the small-world neural network is asymptotically 
stable in the sense of mathematical expectation if the 
number of connections of each neuron n c in the networks 
is n c < 2min{^f}. 
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FIG. 1: Maximum eigenvalue of W w.r.t. p 
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FIG. 2: Maximum eigenvalue of W w.r.t. N 

Figures 1 and 2 show the numerical values of 
Amaz (p, N) as a function of the probability p and the 
number of neurons N. In these figures, for each pair of 
values p and N, X max (p, N) is obtained by averaging the 
results of 20 runs. From the above analysis and the above 
figures, we can see that 

1. for any given value of N > 5, A TOax (p, N) increases 
almost linearly from about 2 to about (N — l)/2 as 
p increases from to 1; 

2. for any given value of p e (0,1], X max {p,N) in- 
creases almost linearly to +oo as JV increases to 
+oo. 

The above results imply that if the given matrix A 
satisfies min{^-} > 2 then 

1. for any given N > 5, there exists a critical value p* 
such that if < p < p* , then the small- world neu- 



ral network is asymptotically stable about its zero 
state (in the sense of mathematical expectation); 
2. for any given p s (0, 1], there exists a critical value 
N* such that if 5 < N < N*, then the small- world 
neural network is asymptotically stable about its 
zero state (in the sense of mathematical expecta- 
tion). 

Clearly, neural networks with small-world connec- 
tions are easier to be stabilized than their regular fully- 
connected counterparts. 

Next, we consider an example of network (1), with the 
constant vector I = 0, A = diag{5, 5, • • • ,5}, and the ac- 
tivation condition </(-)=tanh(-), which also satisfies con- 
dition (3). 

From the above results, we know that for any given 
(or any given p), there exist a corresponding p (or 
a corresponding N) that guarantee the stability of the 
network. The shadow zone in Figure 3 shows the values 
of p and N that ensure the stability of this small-world 
neural network. In this example, the averaged number 
of connections of each neuron in various configurations is 
n c = 9.8592. This result also coincides with the above 
analysis. 
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FIG. 3: Stability zone of the network in the given example 

In summary, a small-world neural network model has 
been presented and its stability has been analyzed. An 
analytical expression, which establishes the relationship 
between the stability and the probability p, has been 
derived. Because there are many biological neural net- 
works that present small- world connections, the results 
obtained in this paper are practical and should be useful 
for further studies of this kind of neural network models. 
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